Performance Evaluation and Experimental Assessment - Conscience or Curse of Database Research?

Performance, performance and performance used to be the three things that really mattered in database research. Most of our published works indeed include an experimental evaluation of the proposed techniques. However, such evaluations are sometimes seen as a ”must-have” eating up the valuable space where one could describe new ideas. The experimental evaluations end up being short, lacking important information to interpret and/or reproduce the results, and often end without clear conclusion. Potential issues of controversy range from the general question, whether experimental assessment and performance evaluation are considered part of research or rather part of engineering, to practical problems such as how experiments should be designed, conducted, documented, and interpreted to make the results reproducible and meaningful for others. The distinguished panelists will discuss their viewpoints on the following questions: